Évaluation des annotations : ses principes et ses pièges

A lot of data is produced by NLP (automatic systems) and for NLP (reference corpus, for computational linguistics or for machine learning) and should be publicly released only if their consistency is proven. While the growing effort that has been made in this direction over the past two decades is encouraging, for example through the increasing use of inter-annotating agreement measures such as kappa, it is not always accompanied by sufficient knowledge of the principles underlying evaluation or the rigor required for their application. The aim of this paper is to present and question the basic concepts and principles of the domain (e.g., shall we use "chance correction" in agreement measures, and if so, how?), and to illustrate with concrete and quantified examples the consequences of an approximate practice of evaluation. MOTS-CLÉS : accord inter-annotateurs, gold standard, évaluation d’annotations.